This specification relates to processing queries, particularly to queries including both an image and associated audio.
The Internet provides access to a wide variety of resources, for example, video files, image files, audio files, or Web pages including content for particular subjects, book articles, or consumer products. A search system can select one or more resources in response to receiving a search query. A search query is data that a user submits to a search engine to satisfy the user's informational needs. The search system selects and scores resources based on their relevance to the search query. The search results are typically ordered according to the scores, and provided in a search results page.
To search image resources, a search system can determine the relevance of an image to a text query based on the textual content of the resource in which the image is located and also based on relevance feedback associated with the image. Some search systems search image resources by using query images as input. A query image is an image, such as a jpeg file, that is used by a search engine as input to a search processing operation. Related images can be found by processing other images and identifying images that are similar in visual appearance to the query image. The use of query images is becoming much more prevalent with the advent of smart phones that include cameras. For example, using a smart phone, a user can now take a picture of a subject of interest, and submit the picture to a search engine. The search engine then searches image resources using the picture as a query image.
However, viewers interpret images in a much more subjective manner than text. Thus, while the images that are identified may be similar in appearance to the query image, many of the images may not be of interest to the viewer. For example, a user may conduct a search on an image of a car. The user may be interested in other cars of that brand, but an image search based only on visual similarity might return images of cars of different brands.